SYSTEMS  ENGINEERING 

Research  Center 

A  US  DoD  University  Affiliated  Research  Center 


Annual SERC 
Research  Review  -  RT  19 

November  9-10,  2010 
Hosts:  University  of  Maryland 
Fraunhofer  Center 
College  Park,  MD 

www.SERCuarc.org 


Form  Approved 
OMB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

NOV  2010 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2010  to  00-00-2010 

4.  TITLE  AND  SUBTITLE 

Annual  SERC  Research  Review  -  RT  19 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Stevens  Institute  of  Technology, Systems  Engineering  Research  Center 
(SERC),1  Castle  Point  on  Hudson, Hoboken, NJ, 07030 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  the  2nd  Annual  SERC  Research  Review  Conference,  9-10  Nov  2010,  College  Park,  MD.  SERC 
is  sponsored  by  the  Department  of  Defense.  U.S.  Government  or  Federal  Rights  License 


14.  ABSTRACT 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

31 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


RT19:  Research  on  Building  Education  & 
Workforce  Capacity  in  Systems 
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Building  Education  &  Workforce  Capacity 

in  Systems  Engineering 

Research  Question 

What  methods,  approaches,  environments,  and  materials  lead  to 
greater  SE  learning,  career  interest,  and  interest  in  DoD  problems? 


Approach 

Develop  course  materials  and  other  value-added  SE  inputs  and 
conduct  pilot  courses  in  14  diverse  institutions;  assess  impacts  on  SE 
learning,  career  interest  and  interest  in  DoD  problems  among 
undergraduate  and  graduate  students. 
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Context  for  Research 


•  Pilot  programs  within  UG/G  capstone  courses,  to  provide 
students  with  substantive  practical  experience  with  SE 
concepts  and  skills,  and  with  opportunities  for  the 
development  of  a  final  product. 

•  Capstone  course  work  will  focus  on  authentic  DoD 
problems,  with  participation  from  DoD  representatives  in 
teams'  projects 


Deliverables 


•  Analysis  of  learning  outcomes  by  principal  investigators 
of  individual  pilot  programs  based  on  progress  as  of 
January  2011  and  June  2011. 

•  Compilation,  synthesis  and  analysis  of  learning 
outcomes  from  the  pilot  programs  for  project  periods 
through  January  2011  and  June  2011. 

•  Recommendations  based  on  the  pilot  programs  that 
will  inform  the  development  of  a  larger  scale-up  effort 
to  build  capacity  for  SE  learning  nationwide. 
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The  Value  of  Systems  Engineering 

Capstone  Courses 

Dr.  Laura  Adolfie 
Director,  STEM  Development  Office 
DDR&E/  Research  Directorate 

2nd  Annual  SERC  Research  Review 
November  9,  2010 
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Systems  Engineering  Mission 


-  Develop  and  grow  the  Systems  Engineering 
capability  of  the  Department  of  Defense 


■  A  robust  Systems  Engineering  capability  across  the 
Department  requires  attention  to  Policy,  People, 
and  Practice 

■  We  apply  best  engineering  practices  to: 

■  Support  and  advocate  for  DoD  Component  initiatives 

■  Help  program  managers  identify  and  mitigate  risks 

■  Shape  technical  planning  and  management 

■  Provide  technical  insight  to  OSD  stakeholders 

■  Identify  systemic  issues  for  resolution  above  the 
program  level 
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Focus  Areas  for  DDR&E:  Improving  the 
Department’s  Engineering  Practice 


-  Growing  the  Systems  Engineering  Workforce 

■  Improving  SE  Capabilities  in  DoD  and  the  Defense  Industrial  Base 

■  Growing  our  Nation's  Future  SE  Workforce 


Our  Focus:  Policy,  People  and  Practice 
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What  Are  Our  Expectations  of  Our 
Systems  Engineering  Workforce? 


■  Breadth 

■  Depth 

■  Leadership 
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We  Need  Systems  Thinkers 
with  Real  World  Context 


■  Today's  technology  systems  are  extremely  complex,  cross- 
disciplinary,  somewhat  non-deterministic,  and  rarely  standalone 

■  We  give  student  opportunities  to  have  an  authentic  DoD 
experience: 

■  Work  on  real  DoD  projects 

■  Interact  with  DoD  and  Defense  Industry  mentors 


21st  CENTURY  SKILLS 


Systems  thinking 


Self-management/ 

Self-development 


Adapability 

Complex 
communication  / 
social  skills 


Nonroutine 
problem  solving 
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Service  Academy  Participation 


■  US  Military  Academy 

■  US  Naval  Academy 

■  US  Coast  Guard  Academy 


■  Air  Force  Institute  of  Technology 
School  manage: 

■  US  Air  Force  Academy 


and 


Naval  Postgraduate 


■  Allows  separate  but  aligned  efforts  with  the  8  civilian 
SERC  schools 

■  Targets  future  engineering  officers  in  the  Services 
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Sponsor  Goals  for  the  Program 


■  Expose  graduating  undergraduate/early  Masters 
students  to  DoD  SE  topics 

■  Test  innovative  ways  to  insert  DoD  content  into 
curricula 

■  Increase  numbers  of  SEs  who  can  enter  DoD 
workforce  quickly 

■  DoD  Laboratory  Enterprise 

■  DoD  Acquisition  Programs  &  Offices 

■  Defense  Contractors 
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Methodology 


Operate  Pilot  at  8  civilian  universities  &  6  service  academies/military  graduate  institutions 
SE  Capstone  course  design 

■  1st  semester  lecture/seminar  course 

■  2nd  semester  project 

Project  Teams  select  from  4  project  areas 

■  Immersive  Training 

■  Low  Cost  Computing 

■  Expeditionary  Green  Housing 

■  Expeditionary  Operations 

Project  Teams  work  with  mentors 

■  DoD 

■  Defense  Primes 

SERC  to  collect  and  report  "Lessons  Learned" 

Program  Deliverables 

■  Syllabi 

■  Progress  Reports 

■  Project  Outcomes/Papers  &  Outbriefs 

■  "Lessons  Learned"  with  recommendations  for  paths  forward 
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SE  Capstone  Student  Projects 


■  Project  descriptions  available  for  review  at: 


http:/  / www.sercuarc.org/ fileadmin/  sercuarc/ SERC  Publications/ 

SE  Capstone  Summaries  All  ASRR  27QctlO  .pdf 
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Sponsor  Focus 


■  Provide  Access  &  Opportunities  that  Build  Student 
Interest 

■  Provide  Mentors  from  Government  &  Defense 
Industry  for  all  Teams 

■  Foster  Long-Term  Relationships 

■  Build  Community  of  Interest  &  Practice 

■  Track  and  Assess  Quality/Quantity  of  Mentor 
Involvement 


2nd  Annual  SERC  Research  Review 
11/09/10  Page- 15 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release  by  OSR  on  30  July  201 0  -  SR  case  number  1 0-S-2849. 


Systems  Engineering: 
Critical  to  DoD  Success 


Innovation,  Speed,  and  Agility 

http://www.  dod.  mil/ddre/stem 
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Project  Schedule 
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Partners 


Civilian  Universities 


1.  Auburn  University 

2.  Missouri  University  S  &  T 

3.  Penn  State 

4.  Southern  Methodist  University 

5.  Stevens  Institute  of  Technology 

6.  University  of  Maryland 

7.  University  of  Virginia 

8.  Wayne  State 


AUBURN 


MISSOURI 

S&T 


Lrn 

Scutum: 


mrrsttyof 
6c  Technology 


Stevens 

INSTITUTE  0/  TECHNOLOGY 
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University 

"RGINIA 


Wayne  State  University 


Service  Academies 


1.  Air  Force  Institute  of  Technology 

2.  Naval  Postgraduate  School 

3.  Air  Force  Academy 

4.  Military  Academy  -  West  Point 

5.  Coast  Guard  Academy 

6.  Naval  Academy 


AFIT 


The  Air  Force  Institute  of  Technology 


A  nAivT^NIIM-U  11\TIAM  A 


West  Point 
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UNITED 
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What  we  hope  to  learn 


•  Common  elements  across  diverse  set  of  projects  that  lead  to 
increased  SE  knowledge,  career  interest,  and  interest  in  DoD 
problems. 

•  Differing  effects  and  outcomes  between  civilian  and  military 
institutions,  undergraduate  and  graduate  students,  other 
differentiators. 

•  Differential  effects  of  DoD  mentor  involvement  (preliminary). 

•  Differential  impacts  and  lessons  learned  from  varied  approaches  to 
recruitment,  course  organization,  team  composition,  etc. 

•  Recommendations  for  core  elements  of  success  for  scaling  up 
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Measurement  of  Student  Educational 

Outcomes 

Required  Common  Assessments 

■  Pre/Post  Survey 

■  Knowledge  of  SE 

■  Interest  in  SE  Careers 

■  Awareness  of  DoD  problem  areas 

■  Pre/Post  Case  Student  Analysis  (Bradley  Fighting  Vehicle) 

■  Growth  in  SE  approach/Analysis  (semantic  analysis) 

■  Weekly  Blog  Posts 

■  Qualitative 

■  Progress  in  level  of  sophistication  of  student  analysis 

■  Final  blog  post-cumulative 
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Measurement  of  Student  Educational 

Outcomes 


Customized  Assessments 

■  Faculty-developed  assessments  unique  to  their 
courses 

■  PI  Evaluation  of  course  effectiveness 

■  PI  Reports  on  DoD  and  Industry  Mentors 
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Preliminary  Data 


■  Student  Recruitment 

■  Problem  Area  Selection 

■  Teams 

■  Role  of  Graduate  Students 

■  DoD  Mentor  Impact 

■  Early  Lessons 


11/1/2010 


SERC 


22 


Most  Schools  Made  the  DoD  Project  a 
Choice  for  Students  Doing  Capstone  Projects 


Student  Recruitment 


■  Why  did  students  choose  to  participate? 

■  Students  interested  in  specific  problem  area 

■  Students  interested  in  working  on  a  project  sponsored  by  the  DoD 


NOTE:  13  Partner  institutions  provided  feedback  on  this  preliminary  PI. 
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Problem  Areas  and  Teams 


■  Problem  Area  Selection  Criteria: 

■  Faculty  members  with  expertise  in  a  particular  area 
were  available  and  interested 

■  Problem  area  considered  to  be  related  to  a  social 
concern  and  would  attract  students 

■  Partner  institutions  had  ongoing  project  in  that  area 

■  Teams: 

■  Wide  range  in  the  number  of  teams  at  each  partner 
institution  -  from  1  to  10,  with  an  average  of  four 

■  Team  size  range  -  2  to  24  students;  most  teams 
formed  by  5  students 
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Involvement  of  Graduate  Students 


•  Undergraduates  benefit  from  graduate  students' 
experience 

•  Serve  as  role  models  and  provide  guidance  to 
undergraduates 

•  "We  also  included  distance  education  graduate  students 
in  our  course.  This  is  valuable  in  that  it  requires  the  team 
to  coordinate  their  work  in  advance,  communicate 
effectively,  and  use  virtual  collaboration  tools" 
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DoD  Mentors'  Impact  to  Date 


On  a  scale  of  1  to  5,  with  1  being  "completely  disagree" 
and  5  being  "completely  agree,"  how  much  do  you  agree  or 
disagree  that  having  a  DoD  mentor  has  had  a  significant 
impact  on... 
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DoD  Mentors'  Impact  to  Date 


On  a  scale  of  1  to  5,  with  1  being  "completely  disagree" 
and  5  being  "completely  agree,"  how  much  do  you  agree  or 
disagree  that  having  a  DoD  mentor  has  had  a  significant 
impact  on... 


Degree  of  Involvement 

Number  of  Schools 

Very  involved 

2 

Somewhat  involved 

5 

We  have  a  mentor,  but  we  have  not  worked  with  him/her  yet 

2 

We  do  not  have  a  mentor  yet 

4 

NOTE:  Timeline  for  coursework  implementation  varies  from  one  school  to  another. 
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DoD  Mentors  Impact  to  Date 

Described  by  Pis 

■  "He  was  able  to  go  over  conceptual  design  review  for  each 
team  and  answered  many  questions  from  students" 

■  "Having  the  students  interact  with  the  sponsors  on  a  weekly 
basis  helps  keep  them  focused  on  the  tasks.  It  also  gives 
students  experience  in  dealing  with  project  reporting" 

■  "Provides  realistic  views  on  the  system,  its  functionality  and 
requirements.  Provides  feedback  and  critique" 
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PI  Suggestions  to  Improve  the 
Mentor  Role  in  the  Future 

■  "The  role  of  the  mentor  should  be  clarified.  In  some  cases  the 
DoD  mentor  serves  as  a  primary  stakeholder  but  in  other 
cases  the  mentor  serves  as  an  advisor  guiding  students  to  a 
solution.  Blurring  these  two  roles  can  lead  to  a  confusion" 

■  "It  should  be  clear  to  mentors  that  the  primary  objective  is  to 
provide  high  quality  learning  experience  for  the  students,  not 
to  develop  a  design  that  will  prove  to  be  useful" 

■  "We  should  have  had  an  assigned  mentor  as  we  were 
submitting  the  proposal  to  guide  us  in  how  we  went  from 
initial  concept  up  through  marketing  to  students  and  product 
research  and  development" 
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Lessons  Learned 


■  Need  for  more  planning  time: 

■  To  review  project  definitions  and  clarify  the  roles  of  the  mentors  and  faculty 

■  Earlier  allocation  of  DoD  mentors 

■  Aligning  schedules  and  academic  requirements  a  cross  departments  has  been 
a  challenge  due  to  the  scope  and  multi-disciplinary  nature  of  the  project.  A 
project  manager  would  be  helpful 

■  Better  planning  for  IRB  approval: 

■  Recommendation  for  a  joint  protocol  and  informed  consent  process 

■  Suggestions  for  current  pilot  program: 

■  Organized  visits  to  outside  facilities,  either  on-site  or  electronic/virtual 

■  Students  meeting  with  mentors  outside  the  gates  of  their  institution 
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Next  Steps 


■  January  2011  Interim  Report  with  preliminary  findings 
and  internal  course  evaluations 

■  June  2011  Final  Reports  and  July  2011  Workshop 

■  Analysis  and  Synthesis  of  Learning  Outcomes  from  the 
Pilot  Programs 

■  Recommendations  based  on  the  Pilot  Programs  that 
will  inform  the  development  of  a  scale-up  effort  to  build 
SE  capacity 

■  Establish  mechanism  to  monitor  student/mentor/ 
faculty  relationships 
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